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Assessment: A Collaborative Model 



by 

Geny McLaughlin, John A. Muftb, Harriott D. Calhoun 
Abstract 

The assessment of educational outcomes has been a major area of interest among policy 
makers in recent years. Those with responsibility for assessment often are drawn from the ranks 
of the faculty and have little knowledge of or interest in institutional research. This paper provides 
a means to distinguish the unique, yet interrelated ways in which assessment and institutional re- 
search serve those requiring information for policy purposes. 

Introduction 

"Implementing a successful assessment program requires a fusion of three quite diflerent ac- 
tivities. First, assessment is about teaching, and neglecting the instructional focus means that there 
is little point to the exercise. Second, assessment is research; consequently, it demands attention 
to classic researchers' questions of method, inference, and ethics. Finally, assessment is an admin- 
istrative activity. Certainty, in the early stages of implementing assessment, organizational issues axe 
paramount, as this stage often involves resolving such questions as incentives and new assignments 
of responsibilities.* 

Ewell's points about assessment do an outstanding job describing the nature of assessment and 
also help explain the complexity in the way assessment is done within institutions. Effective as- 
sessment requires the inclusion of individuals who are experts in the issues of learning and in- 
struction; it requires individuals who are experts in technical methodology; and it involves those 
who are knowledgeable in organizational administration. While the lines and roles are complex and 
are not limited to any given individual within an institution, the tlriree frames of reference often in- 
volve someone in Assessment, someone in Institutional Research, and someone in the institution 
who is an expert in specific technical issues. 

How Outcomes Assessment Works 

Most campus outcomes assessment efforts are driven by external reporting requirements. In- 
itially these have come from state-level governing or coordinating boards like those in Tennessee, 
South Carolina, Georgia, Florida, Colorado, Oklahoma, and Virginia. With regional accrediting 
groups such as the Southern Association of Colleges and Schools (SACS) and North Central As- 
sociation of Colleges and Schools, not to mention disciplinary accrediting gi'oups, adopting out- 
comes assessment requirements for accreditation purposes, more and more colleges and universities 
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are being required to formally address the degree to which they are evaluating student learning. 
While institutions may initiate such efforts based on a shared desire for organizational improve- 
ment, the reality is that normally coUeges and universities do not undertake such programs without 
substantial encouragement from external bodies. 

The types of requirements imposed from the outside often determine to a great extent the 
kinds of information gathered and how that information is reported. In situations where stand- 
ardized tests are the norm and where disciplinary faculty have little input into the process of se- 
lecting the instruments, the outcomes assessment activity becomes synonymous with standardized 
testing in the minds of most students, faculty, and administrators involved. In such instances, a 
testing or assessment center becomes the focal pomX of the process and serves mainly to administer 
the tests and report the results internally and externally. Some technical assistance might be pro- 
vided to academic units in interpretation or, in those cases where the unit itself has some control 
over test content and selection, in test construction. Generally speaking, however, the outcomes 
assessment function is perceived to be tangential to the day-to-day teaching and learning operations 
of the institution. 

In environments where coUeges and universities have more control over what and how they 
report in the outcomes assessment arena, several approaches have been utilized to meet reporting 
requirements while attempting to provide useful information for internal curricular growth and de- 
velopment. The centralized testing center is one model that occurs in more top-down situations, 
but there are also variations on it that allow for more input and involvement from teaching faculty. 
On the other end of the spectrum are outcomes assessment programs that oper?;ce more as brokers 
and consultants, providing minimal direction but substantial support to those within the institution 
who have an interest in studying teaching and learning issues. At research oriented universities with 
substantial graduate programs, much of this kind of research can be conducted by graduate students 
with faculty supervision and direction. The faculty reward systems at community colleges and 
baccalaureate institutions can provide further incentives for faculty from a variety of academic dis- 
ciplines to study outcomes assessment within their areas of specialization. Most outcomes assess- 
ment programs in states allowing flexibility in meeting reporting requirements fall between the two 
extremes of top-down and versus decentralized brokering. Most involve some activities best done 



at the institutional level, such as surveys of student opinion, mixed with other activities wiiich are 
more departmentally driven. The former are necessary to provide institutional level perspectives 
while the latter are more effective in involving facuUy in curricular re-examination and reform. 

Building a Quality Process for Assessn^ent 

The literature on quality offers insights useful to implementing assessment processes. It has 
established the importance of identifying a process model in the creation of a means for continuous 
improvement. One of the more successful such models is the Sheward model of Plan, Do, Check, 
and Act (Imai, 1986). It's application might be as follows: 

• PLAN: Develop a conceptual model for assessment that defines the scope of the effort, de- 
velops a measurement plan, and allocates resources for the assessment. 

• DO: Use the measures to capture and store the data in a reliable maimer, restructure and an- 
alyze the data to focus on the use of the results. 

• CHECK: After each part in the proces'- of assessment, consider the degree to which resuUs 
generalize to the concerns of the institution and the various stakeholders. Tliis relevance comes 
as the results are delivered to the users and interpreted in various reports. ResuUs are used to 
improve the assessment process. 

• ACT: Use resuUs to identify the need to make decisions, clarify the alternatives, make deci- 
sions, and advocate positions. The user needs to determine the correctness of beliefs about 
reality and refme what is still not known. Questions from ACTING go into PLANNING. 

The PDCA cycle described above casts assessment as an information process. A similar model 
selected for this paper is the Information Support Circle which has been specifically developed to 
improve the quality of infoimation and the resulting decision making (McLaughlin and Howard, 
1991). This model represents the specific steps of the PDCA as they relate to information support. 
Planning comes from the development of a belief about what is assessment and how do we measure 
the components. Doing assessment results iii the capture and storage of the data and in the re- 
structuring and analysis of these data into interpretable information. Checking comes from the de- 
livery of the data to the user and determining hs value. Acting is where the user influences the 



situation and increases what is known about outcomes. The results from the use of the information 
includes the determination of modifications to the beliefs about the process and a refinement of the 
next set of questions. 

This ci;cle, shown in Figure 1, also indicates the dependence of process quality on those in 
three roles of SuppUer, Producer, and User and the dependence on the process quality on the 
interactions of those in these three roles. Quality comes from the continuous circling through the 
five functions with the interactions of the roles. The quality is no better that the weakest function. 
This is a major departure from the traditional top-down "get me the following data." 

The roles identified by Ewell have sone link to the three roles in the Circle with the respon- 
sibilities of instruction being a source of activity related to the Supplier, methodology of defming 
outcomes to the Producer, and organizational use of information to the User. It might be tempting 
to assign institutional Research (IR) the Producer role. Outcomes Assessment (OA) the role of 
Supplier, and quibble over who is the User, Yielding to this temptation may well result in disa- 
greements and concerns about whose job is what. 

The way to establish quality assessment is to develop an understanding and a model of what 
needs to be done and then to have a means for getting the tasks done. Table 1 takes the roles of 
OA and IR and looks at the types of issues they might address in each of the five functions. It also 
has a column for some of the assistance which might come from other specialists. 

Steps Which Must Be Taken 

Within the framework of Table 1, the following is a more detailed description of the steps and 
a list of the types of tasks which occur in each step. These tasks and descriptions of the functions 
were compiled from considerations of what is required for quality outcomes assessment from the 
various references and then sorted into the five functions. In addition, the authors rated each task 
as being a tas)' for Institutional Research (I), Outcome's Assessment (O), or a shared responsibility 
for both (B). 
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1. Identify and Measure: 



Assessment should be based on a conceptual model, not only of the program being as- 
sessed, but also of the context in which it exists where the components of the environment are 
related to the anticipated process arid product. This model should include the nature of student 
change in a manner consistent with the objectives, vision, and other chaiacteristics of the in- 
stitution. This conceptual model also identifies the goals as well as the components and con- 
structs of the assessment process. Sunmiative evaluations are especially dependent on the 
identification of goals for the program. In this case, the goal of the assessment is typically to 
measure outcomes which indicate the degi'ee to which the program achieves its goals. If form- 
ative methodology is used, then the evaluation of the process does not have an automatic goal 
but will tend to look for opportunities for improvement. 

Measurement is the identification of the specific measures which will be used in the eval- 
uation. This includes the outcomes as well as the process and where appropriate, the input 
measures. Key issues of the various stakeholders, objectives of the assessment, and likely use 
of the results should be considered. 

a. (B)Begin with what is considered important and is valued. 

b. (B)Start with issues of how assessment results will be used and focus on important things. 

c. (O)Have clear, explicitly stated purpose and goals for assessed programs. 

d. (O)Consider learning to be multidimensional, integrated with participants' beliefs, and 
revealed in performance. 

e. (O)Consider both the outcomes and the processes which produced the outcomes. 

f. (O)Involve a broad range of the educational community in the evaluation. 

g. (I)Include data about longitudinal student enrollment and performance by student type. 

h. (B)lnclude measures about what happens to graduates including participation and per- 
formance in careers. 

i. (O) Include measures about what students know and can do. 

j. (B)Identify level of aggregation based on the audience to whom the report is made and 

the decisions they need to make, 
k. (I)Identify relevant institutional variables already available. 
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I. (O)Solect the variables after reviewing literature, looking at previous, institutional studies, 
and talking to knowledgeable individuals. 

Capture and Store: 

After identification of the items to be measured, the data must be obtained from some 
source and then stored in a manner so that it is secure, accessible, and has documented pro- 
cedures and definitions. Data can come from four basic types of research methodologies 
(Brewer and Hunter, 1989). 

• Field-work where the worker enters natural social groups and studies them, as far as 
possible, in their full and natural state. 

• Survey research which uses questionnaires obtaining data from a large number of indi- 
viduals. 

• Experimentation where the researcher changes specific parts of the situation which is to 
determine causality with the control of various measures and then the manipulations of 
the other measures. 

• Nonreactive reseai'ch which uses the strategy of searcliing out data which already exists 
such as in course grade data bases and opportunities for unobtrusive measurement such 
as news on alumni. 

The best idea is to put the various types of research into combinations, so outcomes as- 
sessment must be prepared to handle all four types of data. More specifically, these methods 
will yield data based on the specific location of the individual at the time of the activity. Pro- 
cedures are necessary to bring the data into the institutional systems and to store it so that 
others can find them and interpret sampling and measurement procedures. This includes 
knowing the data bases which currently exist as well as how to manage data. For example, 
standard categories and codes will need to be developed if career patterns are to be part of the 
data base so that trends will represent changes in the outcomes rather than changes in the 
coding procedure. 

a. (B)Create data systems useful for various groups of individuals; they should be flexible and 
usable in various environments; they should be simple and easy to use. 
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b. (I)Obtain data required from and sometimes develop new student unit record systems, 

c. (I)Require data bases to meet proper standards of data management and administration. 

d. (I)Indude analysis data such as measures of demographic, educational background, initial 
enrollment status descriptors, and institutional activity such as credit hours taken, major, 
etc. 

e. (B)Check for sources for the data and augment where needed with most augmentation 
from the internal design of questionnaires for v.xricus groups of potential students, stu- 
dents, and former students. 

f. (B)Have proper psychometric properties for surveys and send to the proper sample. 

g. (B)Clearly define field studies and various observation methods to remove perceiver bias 
and subjectivity. 

h. (I)Extract relevant data elements from the operating system and put into a cens js file. 

i. (I)Merge data from various niuhiplc academic periods into a single file. 

Restructv*re and Analyse; 

Restructuring and analysis are the means of merging and converting the data into the us- 
ers' frame of reference along with the process of reducing the complexity of the data without 
major losses of relevant detail, They involve integrating various data sets and focusing them 
on the issues of the assessment. The data being integrated and restructured include the tradi- 
tional institutional data such as student characteristics, organizational indices such as applica- 
tions, and developmental data such as from attitude surveys. Restructuring the data moves the 
data from being measures of input into the products and processes of the program which is 
under assessment. Data should be summarized tnd analyzed such that inferences of causality 
can be made with reasonable interpretations. Data reduction occurs by looking only at sub- 
groups, combining variables, and looking at trends. The analysis phase of information support 
reduces the amount of data to something which can be considered by an individual while re- 
taining the major facts in the data. This step concerns the internal validity of the assessment 
and is critical for the interpretation of the causality of the program in producing the outcomes 
being considered. 
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a. (I)Include standard persistence measures such as percentages enrolled at ''normal progress', 
enrolled but not at normal progress, and enrolled but not in good standing. 

b. (J)Include degree-completion indicators such as percentage completing programs, time to 
completion, and percentage of credits required at graduation. 

c. (I)Usc contextual indicators for the institution such as percent attempting normal progress 
and attending continuously. 

d. (B)Usc multivariate procedures and, where possible, adjust for various conditions wliich 
occur while the students are eiaroUed such as changuig the institutional calendar. 

e. (B)Usu a combination of methods and look at the results from several perspectives since 
all methods and measures have limitations in the mterpretation of resuUs. 

f. (B) Analyze the data with a focus on the concerns and the capabilities of the users. 

g. (B)Use statistical procedures which can be replicated by others. 

h. (B)Usc defmitions consistent with external reports, where possible. 

Delivery and reporting: 

The results need to provide user support so that the user can generalize the resuUs of the 
assessment for the various desired uses. The delivery of the results should be structured so that 
the user can check whether the resuhs and data apply to other current situations and/or future 
situations. If the program worked will it work next year? Delivery and reporting should allow 
the user to determine how to use the results in a valid manner. This is particularly critical if 
the uiformation is made available in raw form for user analysis since the user would not have 
the full description and possible admonitions of a report. 

There is a dimension of refinement of the resuhs as provided to the user. At the raw end 
of the dimension are the data themselves, often after some editing and subset selecting and 
merging. Such a delivery would be the student longitudinal fde. At the other end of the re- 
finement of data is the highly refmed, integrated, and interpreted conclusion and recommen- 
dation on the program being considered or the outcome being evaluated. Most of the deliveries 
are between these two extremes with tables, descriptions, some conclusions, and some sug- 
gestions offered for further research. 
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The ability to generalize results comes from checking them against the model noted above 
in the identification and measurement of the activities and process. As noted earlier, this model 
is most likely a set of beliefs about "how things v/ork here'' held by a key administrator md 
shared to varying degrees within the institution culture. In classical methodology tliis gener- 
alization check is the same as the test of tho adequacy of the theory from which wc gain 
knowledge and either change our belief of reality or choose to collect additional data. 

The value of the results comes from theii* appHcability to various students. Results of an 
outcomes assessment can be checked for value based on their: 

a. Adequacy - Mow well do the empirical results support previously held beliefs? 

b. Methodology focus - Do resuU come from the content without methodology bias? 

c. Measurement focus - Were the parts appropriately measured? 

d. Causal inference - Did results support ability to attribute outcomes to specific causes? 

e. Generalization - How well can one generalize the results to other groups? 

f. Realism - Does the simplification of beliefs from the use of available measures adequately 
represent reality? (Brewer, Hunter, 1990), 

If these questions can not be answered afilrmatively, then the mismatch of the assessment and 
the needs of the institution prevent the external validity of the assessment and limit the influ- 
ence of its resuhs in any possible use. The following ai*e some of the steps which are done in 
outcomes assessment to support the delivery of quality resuhs to potential users: 

a. (I)nie dcUvery systems and documentation must be easy to use and at the level of the 
users ability. 

b. (B)Data should be put in a system v/here statistics can be computed in an easy manner. 

c. (B)Reporting is ongoing and evaluated and refined as an ongoing process. 

d. (O) Reporting encourages dealing with real problems and not working just to maximize 
the numbers. 

e. (O)The degree to which the resuhs can be generalized to various groups is determined as 
part of the process. 

f. (B)ResuUs are integrated into the standard information systems of the college. 

g. (O)Results are distributed to the various committees of the college. 



5. Use and Influence: 



A key part of the use and influence of results from assessment is the degree to which the 
results change the way people perceive reality at the institution. The influence of the assess- 
ment rests on the amount of change it makes in the beliefs of individuals. Facts mcrease the 
knowledge of the various participants and will also help identify next steps, unknown compo- 
nents, and data needs. The results enable the user to test and clarify the constructs wliich form 
beliefs about reality, how the process does work, and how it should work. This requires that 
the results be tied back to the constructs of the conceptual model which was used to guide the 
study. Results of the assessment need to be integrated mto the thouglit and decision process 
of the individuals involved in making resource and procedure decisions. 

Some of the major possible uses and influences of assessment results are to: 

a. (O) Demonstrate that responsibilities to the public and students are being met. 

b. (O)Increase emphasis on links between higher education and wider society goals of re- 
newed economic development and global competitiveness. 

c. (O)Move the focus of accountability from equitable access and efficiency to "return on 
investment." 

d. (B)Meet the requirements of laws related to information on students. 

e. (B)Make resuUs part of a larger set of conditions that promote change. 

f. (O) Maintain academic standards and do not "remove" real standards. 

g. (O) Maintain access to higher education. 

h. (O)Keep implementation costs in line. 

i. (O)Provide contextual information that allows conclusions about the factors responsible 
for variations in performance. 

j. (O)Encourage the development of "best practice" at the institutional level in the support 

of research on outcomes, 
k. (O) Build a dialogue around the results of the assessment using short issue oriented studies 

and reports. 

1. (B)Modify student recruitment and retention program development, 
m. (I)Revise student data bases. 
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n. (I) Use in planning and budgeting. 



Developing Cooperation and Coordination 

A summary analysis of the data above produces several quick observations. Of the fifty 
specific activities, the largest of the three categories involves both IR and OA with forty percent 
of the tasks identified as most likely for both functions (B). The next interesting fact is that the 
results of the analysis tends to cast IR in the role of the Supplier and OA in the role of User. 
This differs from the initial belief of most likely roles. At the same time, there is tremendous 
overlap. Effective assessment is dependent on description of duties and cooperative efforts. 

The literature on organizational behavior, group processes, and learning theory speaks to 
the centrality of cooperation and coordination in effective task accomplishments. Outcomes 
assessment in higher education involves complex tasks that receive diverse levels of under- 
standing and commitment by the various segments of the institution. The model proposed here 
is one of collaboration in which the key players on the assessment field (the suppliers, pro- 
ducers, and users of data) maximize the results of their efforts through effective cooperation 
and coordination. 

In synthesizing the experiences of seven institutions involved in improving the use of 
outcomes information in decision making, Kennick (1989) has identified both organizational 
and technical factors that appear to be highly related to the use of assessment results. Briefly, 
these factors pertain to the institution's organizational and managerial environment and to the 
qualities of the assessment information itself. Calhoun (Nichols, 1991) has described the need 
to use both time and persormel effectively in an institutional assessment program. The organ- 
izational environment, technical qualities of the data, and time constraints are all pertinent to 
the effective performance of suppliers, producers, and users of assessment results. 

The following suggestions are directed toward overcoming the organizational, time, and 
personnel constrains that impede use of outcomes information. 

a. Articulate the reason for assessment (external requirements, internal commitment, both). 
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b. Formulate policies for release of results and use of findings for formative and summative 
evaluation. 

c. Establish linkage and dialogue among the suppliers, producers, and users of assessment 
results. 

d. Provide opportunity for meaningful interaction among those who develop, manage, and 
use institutional data bases so that there are numerous opportunities to detect and correct 
system errors. 

e. Identify appropriate assessment activities at the in.stitutional an.d department/program 
levels; determine responsibilities at each level and provide adequate support. 

f. Inventory student outcomes information that is already available and communicate its 
availability throughout the institution. 

g. Establish priorities that insure reasonable access to existing data b"^-'' 

h. Organize information around issues or problems that the institution is committed to ad- 
dressing. 

i. Summarize findings so that the '^bottom line'' is easily reached by those who are less than 
enamored with the detail required by scientific methodology. 

j. Balance thoroughness with timeliness so that adequate information is available to support 

decision making before the opportunity to decide has passed, 
k. Establish baseline information and set targets so that expectations of results are realistic. 
1. Develop a routine mechanism for reviewing results and recommending use. 
m. Integrate use of results with the planning and program review processes of the institution. 

Institutions that assess themselves with appropriate and realistic measures not only have 
information vital for improvement but also the means with which to demonstrate effectiveness 
to others. Implementing a program of assessment makes substantial demands on already 
limited resources of time, personnel, and money. Appropriate coordination and cooperation 
are essential for maximizing the benefits of this commitment. 

In summary, it was the intent of the authors to show that the roles of OA and IR are 
tightly entwined in doing successful assessment. Also, suggestions were provided on how to 
improve the effectiveness of a cooperative method by describing the relationship with a five 
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step model, providing examples of specific tasks which are most likely to be required, and 
finally offering seme suggestions on coordination of effort. These are meant to be descriptive 
in content and process rather than prescriptive in policy or procedure. 
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